Notes here on the following services

• SQS
• SWF
• SNS
• Elastic Transcoder
• Kinesis

 

SQS

The oldest AWS service, first one to go public.

Amazon Simple Queue Service (Amazon SQS) offers a secure, durable, and available hosted queue that lets you integrate and decouple distributed software systems and components. Amazon SQS offers common constructs such as dead-letter queues and cost allocation tags. It provides a generic web services API and it can be accessed by any programming language that the AWS SDK supports.

Features

• Pull based, not push
• Messages are 256KB
• Messages can be kept 1 minute to 14 days
• Visibility Time Out is the amount of time message is invisible in SQS after reader picks it up. Message should be deleted before timeout, else it will become visible and possible to have same message processed twice.
• Visibility Time Out maximum is 12 hours, default is 30 seconds
• SQS guarantees that messages processed at least once
• SQS long polling available. Allows response from queue to wait until there is a message or until the poll times out. This way you save polling cost. Short polling would respond immediately thus the consumer may poll many times incurring cost. Measured in ReceiveMessageWaitTimeSeconds.  The maximum long polling wait time is 20 seconds.

 

How it works

 A producer (component 1) sends message A to a queue, and the message is distributed across the Amazon SQS servers redundantly.

 When a consumer (component 2) is ready to process messages, it consumes messages from the queue, and message A is returned. While message A is being processed, it remains in the queue and isn’t returned to subsequent receive requests for the duration of the visibility timeout.

 The consumer (component 2) deletes message A from the queue to prevent the message from being received and processed again when the visibility timeout expires.

Retention Periods

Amazon SQS automatically deletes messages that have been in a queue for more than maximum message retention period. The default message retention period is 4 days. However, you can set the message retention period to a value from 60 seconds to 1,209,600 seconds (14 days).

Standard Queues

Amazon SQS offers standard as the default queue type. Standard queues support a nearly unlimited number of transactions per second (TPS) per API action (SendMessage, ReceiveMessage, or DeleteMessage). Standard queues support at-least-once message delivery. However, occasionally (because of the highly distributed architecture that allows nearly unlimited throughput), more than one copy of a message might be delivered out of order. Standard queues provide best-effort ordering which ensures that messages are generally delivered in the same order as they’re sent.

FIFO Queues

FIFO (First-In-First-Out) queues are designed to enhance messaging between applications when the order of operations and events is critical, or where duplicates can’t be tolerated, for example:

Amazon MQ is a managed message broker service that provides compatibility with many popular message brokers. We recommend Amazon MQ for migrating applications from existing message brokers that rely on compatibility with APIs such as JMS or protocols such as AMQP, MQTT, OpenWire, and STOMP.

SQS and MQ are a pull based messaging service. SNS does push.

 

SWF

The Amazon Simple Workflow Service (Amazon SWF) makes it easy to build applications that use Amazon’s cloud to coordinate work across distributed components. In Amazon SWF, a task represents a logical unit of work that is performed by a component of your workflow. Coordinating tasks in a workflow involves managing intertask dependencies, scheduling, and concurrency in accordance with the logical flow of the application.

Tasks in SWS can sit in the queue for 1 year. SWF is task-oriented, whereas SQS was message-oriented. SWF ensures that a task is assigned only once and is never duplicated, whereas SQS may have duplicate message based on how quickly it processes it.

Actors

In the course of its operations, Amazon SWF interacts with a number of different types of programmatic actors. Actors can be workflow starters, deciders, or activity workers. These actors communicate with Amazon SWF through its API. You can develop these actors in any programming language.

Domain

Domains provide a way of scoping Amazon SWF resources within your AWS account. All the components of a workflow, such as the workflow type and activity types, must be specified to be in a domain. It is possible to have more than one workflow in a domain; however, workflows in different domains can’t interact with each other.

When setting up a new workflow, before you set up any of the other workflow components you need to register a domain if you have not already done so.

When you register a domain, you specify a workflow history retention period. This period is the length of time that Amazon SWF will continue to retain information about the workflow execution after the workflow execution is complete.

 

SNS

Amazon Simple Notification Service (Amazon SNS) is a web service that coordinates and manages the delivery or sending of messages to subscribing endpoints or clients. In Amazon SNS, there are two types of clients—publishers and subscribers—also referred to as producers and consumers. Publishers communicate asynchronously with subscribers by producing and sending a message to a topic, which is a logical access point and communication channel. Subscribers (that is, web servers, email addresses, Amazon SQS queues, AWS Lambda functions) consume or receive the message or notification over one of the supported protocols (that is, Amazon SQS, HTTP/S, email, SMS, Lambda) when they are subscribed to the topic.

Features

• Instant push based delivery
• Simple APIs
• Flexible message delivery over multiple transport protocols
• Inexpensive
• Involves Publishers, Subscribers (and readers – like Email/SMS)
• Once a message is posted to topic, it cannot be recalled or deleted

Delivery Status

Amazon SNS provides support to log the delivery status of notification messages sent to topics with the following Amazon SNS endpoints:

• Application
• HTTP
• Lambda
• SQS

Delivery Retries

Amazon SNS defines a delivery policy for each delivery protocol. The delivery policy defines how Amazon SNS retries the delivery of messages when server-side errors occur (when the system that hosts the subscribed endpoint becomes unavailable). When the delivery policy is exhausted, Amazon SNS stops retrying the delivery and discards the message—unless a dead-letter queue is attached to the subscription.

Dead-Letter Queues

A dead-letter queue is an Amazon SQS queue that an Amazon SNS subscription can target for messages that can’t be delivered to subscribers successfully. Messages that can’t be delivered due to client errors or server errors are held in the dead-letter queue for further analysis or reprocessing

 

Elastic Transcoder

Amazon Elastic Transcoder lets you convert media files that you have stored in Amazon Simple Storage Service (Amazon S3) into media files in the formats required by consumer playback devices. For example, you can convert large, high-quality digital media files into formats that users can play back on mobile devices, tablets, web browsers, and connected televisions.

Elastic Transcoder has four components:

• Jobs
  • Does work of transcoding
  • Can convert up to 30 formats
• Pipelines
  • Queues that manage the transcoding jobs
• Presets
  • Templates that define settings for transcoding
• Notifications

 

Kinesis

You can use Amazon Kinesis Data Streams to collect and process large streams of data records in real time. You can create data-processing applications, known as Kinesis Data Streams applications. A typical Kinesis Data Streams application reads data from a data stream as data records. These applications can use the Kinesis Client Library, and they can run on Amazon EC2 instances.
The Kinesis Services are:

• Kinesis Data Streams
• Kinesis Data Firehose
• Kinesis Data Analytics
• Kinesis Video Streams

Kinesis Data Streams

Amazon Kinesis Data Streams ingests a large amount of data in real time, durably stores the data, and makes the data available for consumption. The unit of data stored by Kinesis Data Streams is a data record. A data stream represents a group of data records. The data records in a data stream are distributed into shards.

A shard has a sequence of data records in a stream. When you create a stream, you specify the number of shards for the stream. The total capacity of a stream is the sum of the capacities of its shards. You can increase or decrease the number of shards in a stream as needed. However, you are charged on a per-shard basis.

A single shard can ingest up to 1 MB of data per second (including partition keys) or 1,000 records per second for writes. Similarly, if you scale your stream to 5,000 shards, the stream can ingest up to 5 GB per second or 5 million records per second.

The retention period is the length of time that data records are accessible after they are added to the stream. A stream’s retention period is set to a default of 24 hours after creation. You can increase the retention period up to 168 hours (7 days).

Kinesis Firehose

Amazon Kinesis Data Firehose is a fully managed service for delivering real-time streaming data to destinations such as Amazon Simple Storage Service (Amazon S3), Amazon Redshift, Amazon Elasticsearch Service (Amazon ES), and Splunk. Kinesis Data Firehose is part of the Kinesis streaming data platform, along with Kinesis Data Streams, Kinesis Video Streams, and Amazon Kinesis Data Analytics.

Firehose has no retention period, the incoming data needs to be processed realtime or stored to S3. There are no shards, it is processing the data records directly. Each record can be up to 1000KB.

Kinesis Data Analytics

Amazon Kinesis Data Analytics analyze streaming data, gain actionable insights, and respond to your business and customer needs in real time. Amazon Kinesis Data Analytics reduces the complexity of building, managing, and integrating streaming applications with other AWS services. SQL users can easily query streaming data or build entire streaming applications using templates and an interactive SQL editor.

Kinesis Video Streams

Amazon Kinesis Video Streams is a fully managed AWS service that you can use to stream live video from devices to the AWS Cloud, or build applications for real-time video processing or batch-oriented video analytics.

Kinesis Data Stream vs Firehose

Kinesis Data Stream also stores the data in the stream until its processed. The underlying stream is also configurable on how to setup the shards.

Firehose automatically delivers the data, there is no stream storage area like Data Stream.

 

 

 

References

SQS AWS DOCS
https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/

SNS AWS DOCS
https://docs.aws.amazon.com/sns/latest/dg/welcome.html

KINESIS
https://docs.aws.amazon.com/kinesisvideostreams/latest/dg/what-is-kinesis-video.html